TinyViT: Fast Pretraining Distillation for Small Vision Transformers

Vision transformer (ViT) recently has drawn great attention in computer vision due to its remarkable model capability. However, most prevailing ViT models suffer from huge number of parameters, restricting their applicability on devices with limited resources. To alleviate this issue, we propose TinyViT, a new family tiny and efficient small transformers pretrained large-scale datasets our proposed fast distillation framework. The central idea is transfer knowledge large ones, while enabling get the dividends massive pretraining data. More specifically, apply during for transfer. logits teacher are sparsified stored disk advance save memory cost computation overheads. student automatically scaled down parameter constraints. Comprehensive experiments demonstrate efficacy TinyViT. It achieves top-1 accuracy 84.8% ImageNet-1k only 21M being comparable Swin-B ImageNet-21k using 4.2 times fewer parameters. Moreover, increasing image resolutions, TinyViT can reach 86.5% accuracy, slightly better than Swin-L 11% Last but not least, good ability various downstream tasks. Code available at https://github.com/microsoft/Cream/tree/main/TinyViT .